3D virtual environment interaction system

ABSTRACT

There is provided a 3D virtual environment interaction system comprising: a processing unit for generating a 3D virtual environment comprising a planar reference frame for allowing a user to perform a user interaction with the 3D virtual environment; a display unit for displaying the generated 3D virtual environment; a portable input device for allowing the user to control the position and orientation of the planar reference frame within the generated 3D virtual environment, the portable input device comprising a planar input surface; a position and orientation sensor for monitoring the position and orientation of the planar input surface in a real-world space, the position and orientation sensor allowing the processing unit to modify at least one of the position and the orientation of the planar reference frame in response to a change in a corresponding one of the position and orientation of the planar input surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority 35 U.S.C. § 119 of U.S. ProvisionalPatent Application No. 62/013,299, filed on Jun. 17, 2014, thespecification of which is incorporated herein by reference.

TECHNICAL FIELD

The invention relates to 3D virtual environment, and more specificallyto interaction systems and methods for 3D virtual environments.

BACKGROUND

Three-dimensional (3D or 3d) virtual environments are used in a varietyof applications. They have become a mainstay in computer-aided design(CAD) by reducing or even eliminating the need to produce costlyprototypes when designing a new object, part or machine. 3D virtualenvironments can also be used to visualize existing objects andenvironments, for example in immersive displays or flight simulators, orto create new environments such as those found in video games oranimated movies.

To create new content for a 3D virtual environment, edit existingcontent of the environment or simply to navigate through theenvironment, it is necessary for the user to perform interactions (i.e.to interact) with the environment. User interaction in these 3D virtualenvironments can be achieved in a variety of ways. A first methodconsists in inputting commands to a program generating the 3D virtualenvironment through a keyboard. The program will usually have apreprogrammed set of instructions to which it will respond. The user mayinput a particular command corresponding to a certain interaction andmay also input a particular set of coordinates corresponding to thelocation where the interaction is to take place. The interaction couldcomprise drawing lines, vertices, solid objects, etc. within the 3Denvironment, or selecting one or more objects within the 3D environmentto set or change properties of said object or to change its state withinthe 3D environment.

Unfortunately, this method is fastidious and time-consuming, andrequires from the user a precise knowledge of the location where theinteraction will be performed.

This method has been largely superseded by the use of a pointer orcursor which allows the user to indicate where in the 3D virtualenvironment a certain interaction is to take place. The cursor istypically controlled by a computer mouse which is moved on a planarsurface. Unfortunately, this configuration only allows the cursor to bemoved in a static two-dimensional workplane which is generally coplanarwith the screen of a display unit. To select an object or a surface, theuser must bring said object or surface within the workplane, which mayinvolve manipulations within the 3D environment such as hiding otherobjects, modifying the point of view, etc., all of which are relativelycomplex and time-consuming.

There is therefore a need for a new system which would enable simple andefficient interactions within a 3D virtual environment.

BRIEF SUMMARY

According to one aspect, there is provided a 3D virtual environmentinteraction system comprising: a processing unit for generating a 3Dvirtual environment, the 3D virtual environment comprising a planarreference frame for allowing a user to perform a user interaction withthe 3D virtual environment; a display unit operatively connected to theprocessing unit for displaying the generated 3D virtual environment; aportable input device for allowing the user to control the position andorientation of the planar reference frame within the generated 3Dvirtual environment, the portable input device comprising a planar inputsurface; a position and orientation sensor for monitoring the positionand orientation of the planar input surface in a real-world space, theposition and orientation sensor being operatively connected to theprocessing unit to allow the processing unit to modify at least one ofthe position and the orientation of the planar reference frame inresponse to a change in a corresponding one of the position andorientation of the planar input surface.

In one embodiment, the user interaction comprises inserting atwo-dimensional drawing on the planar reference frame.

In one embodiment, the planar input surface is adapted to allow the userto draw the two-dimensional drawing thereon.

In one embodiment, the portable input device comprises at least onevirtual drawing tool selected from a group consisting of: a line drawingtool and a two-dimensional shape drawing tool.

In one embodiment, the processing unit is adapted to produce athree-dimensional drawing when the portable device is moved in thereal-world space in a direction normal to the planar input surface,thereby moving the planar reference frame with the insertedtwo-dimensional drawing.

In one embodiment, the user interaction comprises selecting a virtualobject in the 3D virtual environment.

In one embodiment, selecting the virtual object comprises positioningthe planar reference frame such that the planar reference frameintersects the virtual object.

In one embodiment, the user interaction further comprises scaling theselected virtual object.

In one embodiment, the scaling comprises isotropic scaling.

In one embodiment, the scaling comprises anisotropic scaling.

In one embodiment, the user interaction further comprises moving theselected object.

In one embodiment, moving the selected object comprises at least one ofmoving the selected object in translation and rotating the selectedobject.

In one embodiment, the user interaction further comprises duplicatingthe selected object.

In one embodiment, the user interaction comprises importing an existingvirtual object into the 3D virtual environment, and positioning andorienting said existing virtual object within the 3D virtual environmentaccording to the position and orientation of the planar reference frame.

In one embodiment, the processing unit is adapted to move the planarreference frame according to a selected one of at least one framemovement mode in response to a movement of the planar input surface.

In one embodiment, the at least one frame movement mode comprises a freemovement mode in which the planar reference frame is moved such that theposition and orientation of the planar reference frame within the 3Dvirtual environment corresponds to the position and orientation of theplanar input surface in the real-world space.

In one embodiment, the at least one frame movement mode comprises aplanar movement mode in which movement of the planar reference frame isrestricted to translation in a direction tangential relative to theplanar reference frame.

In one embodiment, the at least one frame movement mode comprises anormal movement mode in which movement of the planar reference frame isrestricted to translation along a linear movement path extending indirection normal to the planar reference frame.

In one embodiment, the at least one frame movement mode comprises ahinged movement mode in which movement of the planar reference frame isrestricted to rotation about a rotation axis extending in a directiontangential relative to the planar reference frame.

In one embodiment, the at least one frame movement mode comprises anangle snapping mode in which the planar reference frame is adapted tomove to one of a plurality of predetermined orientations when theorientation of the planar reference frame is within a predeterminedangular range from said one of the plurality of predeterminedorientations.

In one embodiment, the predetermined angular range comprises a 5-degreerange.

In one embodiment, the planar input surface is further adapted toreceive at least one touch gesture and wherein the processing unit isadapted to modify at least one of the position and orientation of theplanar reference frame in response to said at least one touch gesture.

In one embodiment, the touch gesture comprises a multi-touch gesture.

In one embodiment, the portable input device further comprises asecondary display for displaying the 3D virtual environment.

In one embodiment, the secondary display is adapted to display anorthogonal projection of the 3D virtual environment as projected on theplanar reference frame.

In one embodiment, the display unit is adapted to display the 3D virtualenvironment in a perspective view from a main point of view and thesecondary display is adapted to display the 3D virtual environment in aperspective view from a secondary point of view which is different fromthe main point of view.

In one embodiment, the planar input surface comprises a touchscreen, thetouchscreen further defining the secondary display.

In one embodiment, the display unit is adapted to display the 3D virtualenvironment from a main point of view, and further wherein theprocessing unit is adapted for allowing the user to modify at least oneof the position and orientation of the main point of view.

In one embodiment, the position of the main point of view is movable bysliding a finger on the planar input surface.

In one embodiment, the planar input surface is adapted to receive arotation gesture, the processing unit being adapted to change theorientation of the main point of view according to said rotationgesture.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a 3D virtual environment interactionsystem, in accordance with one embodiment;

FIG. 2 is a portable input device and of a 3D virtual environment forthe 3D virtual environment interaction system shown in FIG. 1;

FIG. 3 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing movement of a planar reference frame in a freemovement mode;

FIG. 4 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing movement of the planar reference frame in aplanar movement mode;

FIG. 5 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing movement of the planar reference frame in anormal movement mode;

FIG. 6 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing movement of the planar reference frame in ahinged movement mode;

FIG. 7 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing a two-dimensional drawing being inserted in theplanar reference frame;

FIG. 8 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing a three-dimensional drawing being drawn in the3D virtual environment;

FIG. 9A is a schematic view of a 3D virtual environment for the 3Dvirtual environment interaction system shown in FIG. 1, showing a firsttwo-dimensional rectangular shape being drawn in the 3D virtualenvironment in a first plane;

FIG. 9B is a schematic view of a 3D virtual environment for the 3Dvirtual environment interaction system shown in FIG. 1, showing a secondtwo-dimensional rectangular shape being drawn in the 3D virtualenvironment in a second plane;

FIG. 9C is a schematic view of a 3D virtual environment for the 3Dvirtual environment interaction system shown in FIG. 1, showing athree-dimensional rectangular prism comprising the first and secondtwo-dimensional rectangular shapes drawn in the 3D virtual environment;

FIG. 10 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing the planar reference frame being moved towardsa virtual object;

FIG. 11 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing the planar reference frame intersecting thevirtual object;

FIG. 12 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing the planar reference frame intersecting thevirtual object and the virtual object moved in rotation and translation;and

FIG. 13 is a schematic view of a portable input device and of a 3Dvirtual environment for the 3D virtual environment interaction systemshown in FIG. 1, showing an orthogonal projection of the 3D virtualenvironment in the secondary display.

DETAILED DESCRIPTION

FIGS. 1 and 2 show a tridimensional (3D) virtual environment interactionsystem 100, in accordance with one embodiment. In this embodiment, the3D virtual environment interaction system 100 comprises a processingunit 102 for generating a 3D virtual environment 104 and a display unit106 operatively connected to the processing unit 102 for displaying thegenerated 3D virtual environment 104.

In one embodiment, the processing unit 102 comprises a personal computerwhich includes a dedicated 3D modeling program which is adapted togenerate the 3D virtual environment 104.

Alternatively, the processing unit 102 may include a personal computerwhich includes an existing 3D modeling program which is adapted togenerate the 3D virtual environment 104. In this embodiment, theexisting 3D modeling program may include a dedicated module which couldbe installed within the existing 3D modeling program in order to enablethe existing 3D modeling program to be used with the system 100described herein. This may render the system 100 more accessible byeliminating the need for the user to acquire an entirely new 3D modelingprogram.

The system 100 further comprises a portable input device 108 whichallows a user to interact with the 3D virtual environment 104.Specifically, the portable input device 108 comprises a planar inputsurface 110 and the 3D virtual environment 104 comprises a planarreference frame 112 associated with the planar input surface 110. Theplanar reference frame 112 acts as a “3D cursor” to allow the user toperform user interactions with the 3D virtual environment 104, and theposition and orientation of the planar reference frame 112 within the 3Dvirtual environment 104 is controlled by the portable input device 108.

To enable this control, the 3D virtual environment interaction system100 further comprises a sensor 114 operatively connected to theprocessing unit 102 for monitoring the orientation and position of theplanar input surface 110. This allows the processing unit 102 to modifyat least one of the position and the orientation of the planar referenceframe 112, and thereby move the planar reference frame 112 within the 3Dvirtual environment 104, in response to a change in a corresponding oneof the position and orientation of the planar input surface 110.

In the embodiments illustrated in FIGS. 2 to 14, the planar referenceframe 112 is rectangular, and the planar input surface 110 is alsorectangular. Alternatively, both the planar reference frame 112 and theplanar input surface 110 could have any other planar shape that askilled addressee may deem fit for the intended use of the system 100.

In one embodiment, the portable input device 108 comprises a tabletcomputer having a touchscreen which defines the planar input surface110. The use of a touchscreen can allow the user to readily have accessto a plurality of functions from the portable input device 108, as willbecome apparent below. Alternatively, the portable input device 108could comprise a smartwatch, a smartphone or any other portable inputdevice known to a skilled addressee.

In one embodiment, the portable input device 108 is wirelessly connectedto the processing unit 102. This allows a relatively large of motion forthe user to move the portable input device 108 into the real-worldspace. Alternatively, the portable input device 108 could be connectedto the processing unit 102 via a wired connection.

In one embodiment, the sensor 114 comprises a magnetic sensor.Specifically, the magnetic sensor comprises a receiver and an emitter.One of the receiver and the emitter is attached to the portable inputdevice 108 and the other one of the receiver and the emitter ispositioned at a static position. The magnetic sensor can thereby monitormovement of the emitter relative to the receiver, which can then beconverted by the processing unit 102 to movement of the planar referenceframe 112 in the 3D virtual environment.

In one embodiment, the magnetic sensor is adapted to monitor both theposition and the orientation of the portable input device 108.Alternatively, the magnetic sensor could be adapted to monitor only theposition of the portable input device 108.

In one embodiment, the portable input device 108 comprises anorientation sensor mounted on the portable input device 108.Specifically, the orientation sensor can comprise at least one of agyroscope and an accelerometer. Orientation of the portable input device108 could therefore be monitored by the portable input device 108itself.

In one embodiment, the sensor 114 comprises an optical sensor.Specifically, the optical sensor could comprise one or more opticalmarkers attached to the portable input device 108 and at least onecamera operatively connected to the processing unit 102 for monitoringthe position and orientation of the portable input device 108 based onthe position of the markers in the real-world space.

In one embodiment, the sensor 114 could comprise a plurality of emittersand receivers disposed to permit the monitoring of the position of theportable input device 108 by triangulation. Specifically, the portableinput device 108 could be adapted to receive and emit a signal (using aBluetooth protocol for example) and the position and orientation of theportable input device 108 would be determined using signals from otheremitters disposed at known locations in the real-world space.

In one embodiment, the sensor 114 comprises a camera mounted to orembedded in the portable input device 108. The camera could be adaptedto determine the position and orientation of the portable input device108 based on a detection of one or more markers disposed at fixedlocations in the real-world space.

It will be appreciated that the sensor 114 is not limited to the sensorsdescribed hereinabove, and that the sensor 114 could instead compriseone or more of various position and orientation sensors known to askilled addressee.

In one embodiment, the display unit 106 comprises a display monitor,such as a computer monitor or a television monitor. Alternatively, thedisplay monitor 106 could comprise a projector and a screen on which the3D virtual environment is displayed.

In one embodiment, the display unit 106 provides an immersive display.Specifically, the display unit 106 could comprise a projector, aspherical mirror and a concave, spherical screen which allows the userto stand at or near its center. Alternatively, the display unit 106could instead comprise a head-mounted display, or any other types ofdisplay known to a skilled addressee.

Now referring to FIGS. 3 to 7, the processing unit 102 may be adapted tomove the planar reference frame 112 according to a desired framemovement mode which is selected by the user from one of a plurality ofavailable frame movement modes. The selection could be made directly bythe user on the planar input surface 110.

In one embodiment illustrated in FIG. 3, the plurality of frame movementmodes includes a free movement mode which allows the user to freely movethe planar reference frame 112 in the 3D virtual environment 104 bymoving the portable input device 108 in the real-world. In this case,movement of the position and orientation of the planar reference frame112 directly corresponds to movement of the position and orientation ofthe planar input surface 110 of the portable input device 108.Specifically, the orientation of the planar reference frame 112 couldcorrespond to the actual orientation of the planar input surface 110 inthe real-world space. The position of the planar reference frame 112could be computed using the sensed position of the planar input surface110.

The plurality of frame movement modes could further include a pluralityof movement restriction modes which can allow the user to better andmore easily position the planar reference frame 112 within the 3Dvirtual environment 104.

In one embodiment illustrated in FIG. 4, the plurality of frame movementmodes could include a planar movement mode, in which movement of theplanar reference frame 112 is restricted to translation in a directiontangential (i.e. coplanar) relative to the planar reference frame 112.In this mode, the planar reference frame 112 will not move in adirection normal to the planar reference frame 112, even if the portableinput device 108 is moved in a direction normal to the planar inputsurface 110. Furthermore, this mode allows the planar reference frame112 to move in rotation only about a rotation axis extending in a normaldirection relative to the planar reference frame 112.

In one embodiment illustrated in FIG. 5, the at least one frame movementmode could include a normal movement mode in which movement of theplanar reference frame 112 is restricted to translation along a linearmovement path extending in a direction normal to the planar referenceframe 112. In this mode, the planar reference frame 112 will not move ina tangential or coplanar direction relative to the planar referenceframe 112, even if the portable input device 108 is moved in a directiontangent to the planar input surface 110. Furthermore, this mode alsodoes not allow the planar reference frame 112 to move in rotation, andtherefore does not allow the orientation of the planar reference frame112 to be modified.

In one embodiment illustrated in FIG. 6, the at least one frame movementmode comprises a hinged movement mode in which movement of the planarreference frame 112 is restricted to rotation about a rotation axis Rextending tangentially relative to the planar reference frame 112.Specifically, the rotation axis R could be defined at the center of theplanar reference frame 112, as shown in FIG. 6. Alternatively, therotation axis R could instead be located at an edge of the planarreference frame 112 to allow rotation of the planar reference frame 112about its edge.

In the embodiment illustrated in FIG. 6, the rotation axis R furtherextends vertically. Alternatively, the rotation axis R could insteadextend horizontally. In another embodiment, the rotation axis R couldinstead extend in any orientation selected by the user. In yet anotherembodiment, the rotation axis R could extend in one of a vertical andhorizontal direction depending on a direction of a rotation axis aboutwhich the user rotates the portable input device when in the hingedmovement mode. If the user rotates the portable input device around avertical axis, then the rotation axis R of the planar reference frame112 also extends vertically. If the user rotates the portable inputdevice 108 around a horizontal rotation axis, then the rotation axis Rof the planar reference frame 112 becomes horizontal as well.

In one embodiment, rotation of the portable input device 108 directlycorresponds to the rotation of the planar reference frame 112. Forexample, a rotation of the portable input device 108 of 90 degreesclockwise about a vertical rotation axis will cause a rotation of theplanar reference frame 112 of 90 degrees clockwise about a verticalrotation axis.

In one embodiment, the at least one frame movement mode comprises anangle snapping mode in which the planar reference frame 112 is allowedto be rotated freely until the orientation of the planar reference frame112 is near certain predetermined angles. Specifically, when in theangle snapping mode, the planar reference frame 112 is adapted to moveto one of the plurality of predetermined orientations when theorientation of the planar reference frame 112 is within a predeterminedangular range from said one of the plurality of predeterminedorientations. This predetermined angular range could for examplecomprise a 5-degree range. Therefore, when the portable input device 108is rotated such that the orientation of the planar reference frame 112comes within 5 degrees of a predetermined orientation, the planarreference frame 112 will immediately rotate (i.e. snap) to thispredetermined orientation.

The planar input surface 110 could further be adapted to receive atleast one touch gesture, specifically in an embodiment in which theplanar input surface 110 is a touchscreen. In this embodiment, theprocessing unit 102 is adapted to modify at least one of the positionand orientation of the planar reference frame 112 in response to said atleast one touch gesture. For example, the at least one touch gesture maycomprise sliding a finger in a generally straight line on the planarinput surface 110 to move the planar reference frame 112 tangentiallyrelative to the planar reference frame 112. In another example, theplanar reference frame 112 could be moved in the normal movement modewherein sliding a finger in a generally straight line on the planarinput surface 110 move the planar reference frame 112 in a normaldirection relative to the planar reference frame 112 by a distanceproportional to the distance travelled by the finger on the planar inputsurface 110.

It will be appreciated that the touch gesture could comprise amulti-touch gesture (i.e. a touch gesture performed with more than onefinger).

Other touch gestures could also be used to perform other functionsrelative to the planar reference frame 112. In one embodiment,multi-touch gestures could further be used for increase and reducing thesize of the planar reference frame 112. Specifically, a pinch gesture,using two fingers, could be used for this purpose. The user would bringthe two fingers closer together to reduce the size of the planarreference frame 112 and would move them further apart to increase thesize of the planar reference frame 112. It will be appreciated that avariety of other touch gestures could be used.

It will also be appreciated that the combination of movement of theplanar reference frame 112 according to the position and orientation ofthe planar input surface 110 and according to at least one touch gesturecould provide the user with better control over the positon andorientation of the planar reference frame 112 within the 3D virtualenvironment.

Now turning to FIG. 7, the system 100 further allows the user performvarious types of user interactions with the 3D virtual environment byusing the planar reference frame 112. In one embodiment, the userinteraction comprises inserting a two-dimensional drawing 700 on theplanar reference frame 112. Specifically, the planar input surface 112could be adapted to allow the user to draw the two-dimensional drawing700 directly thereon. The two-dimensional drawing 700 would thereby bereceived by the portable input device 108 and transmitted to theprocessing unit, which could then insert the two-dimensional drawing 700on the planar reference frame 112. It will be understood that once thetwo-dimensional drawing 700 has been inserted in the 3D virtualenvironment on the planar reference frame 112, it may be fixed at thislocation such that further movement of the planar reference frame 112will not move the two-dimensional drawing 700.

In an embodiment in which the planar input surface 110 comprises atouchscreen, and the two-dimensional drawing 700 could for example besimply drawn or sketched freehand using a finger or a stylus on thetouchscreen. In one embodiment, the portable input device 108 comprisesan input program which includes one or more drawings tools. Thesedrawings tools could be similar to the drawing tools found in aconventional two-dimensional computer-aided design (CAD) program andcould include, for example, a line drawing tool to allow the user toconveniently draw straight lines and a two-dimensional shape drawingtool to allow the user to conveniently draw a plurality of predeterminedtwo-dimensional shapes. It will be appreciated that the input programcould further include any other drawing tool known to a skilledaddressee, including both raster drawing tools and vector drawing tools.

It will be appreciated that the term “two-dimensional drawing” is usedherein to refer to a drawing drawn on the planar input surface 110 andcould also include a “one-dimensional drawing”, i.e. a single point.

In one embodiment, the planar reference frame 112 may be temporarilyfixed at a desired location within the 3D virtual environment. Thiswould facilitate the drawing on the planar input surface 110 by theuser, especially when the planar input surface is oriented at aninconvenient angle.

Now referring to FIG. 8, the system 100 further allows the user to drawa three-dimensional drawing within the 3D virtual environment 104. Toachieve this, the user could start by drawing a two-dimensional drawingon the planar input surface 110. For example, the user could place afinger or a stylus on the planar input surface to create a point 800 onthe planar reference frame. The two-dimensional drawing would then beinserted in the 3D virtual environment 104, at the current location ofthe planar reference frame.

The user could then move the planar reference frame away from thedrawing plane along a drawing path 802. Since the finger or a stylus ofthe user is still in the planar input surface 110, a line 804 would bedrawn along the drawing path, thereby forming a three-dimensionaldrawing.

In one embodiment, the two-dimensional drawing could instead comprise atwo-dimensional shape and the movement of the two-dimensional shapecould form an extrusion having a constant cross-section corresponding tothe two-dimensional shape. Alternatively, the cross-section of theextrusion could be modified by the user as the two-dimensional shape isbeing extruded. For example, the user could make a pinch gesture on theplanar input surface 110 as the portable input device 108 is moved. Thiswould provide a relatively easy and efficient way to create extrusionsin a 3D virtual environment.

It will be appreciated that the two-dimensional drawing is not limitedto a single point or a single shape. The two-dimensional drawing couldcomprise multiple points or shapes, which would respectively createmultiple lines similar to line 804 or multiple extrusions when theplanar reference frame 112 is moved in the 3D virtual environment.

Now referring to FIGS. 9A to 9C, it will also be appreciated thatthree-dimensional drawings could also be created by successively movingthe planar reference frame 112 and drawing multiple two-dimensionaldrawings at different orientations within the 3D virtual environment104.

In one embodiment, the planar reference frame 112 is first oriented in avertical orientation and a first two-dimensional rectangular shape 900is drawn by the user in the 3D virtual environment, as shown in FIG. 9A.The planar reference frame 112 is then rotated, for example using thehinged movement mode as described above, to a horizontal orientation andmoved in translation to allow the user to draw in the 3D virtualenvironment 104 a second two-dimensional rectangular shape 902 which isorthogonal and adjacent to the first two-dimensional rectangular shape900, as shown in FIG. 9B. By successively moving the planar referenceframe 112 to appropriate locations and orientations, the user canthereby draw a three-dimensional rectangular prism 904 within the 3Dvirtual environment 104, the three-dimensional rectangular prism 904comprising a plurality of two-dimensional rectangular shapes 900, 902drawn in different planes.

It will be appreciated the rectangular prism described above is merelyprovided as an example, and that the user could draw numerous differentthree-dimensional drawings using this technique.

Now referring to FIGS. 10 to 12, apart from drawing in the 3D virtualenvironment 104, the user interaction could also comprise selecting avirtual object 1000 on the 3D virtual environment 104. Specifically, the3D virtual environment 104 could comprise one or more discrete virtualobjects which could be manipulated within the 3D virtual environment104.

In one embodiment, selecting the virtual object 1000 comprisespositioning the planar reference frame 112 such that the planarreference frame 112 intersects the virtual object 1000, as shown in FIG.11. Once the planar reference frame 112 intersects the virtual object1000, the user can select the virtual object 1000 via the portable inputdevice 108. One or more virtual objects could also be selected byactivating a selection mode and moving the planar reference frame 112 inthe 3D virtual environment 104. In this mode, every virtual object whichis intersected by the moving planar reference frame 112 is selected.

Selected virtual objects can then be manipulated by the user. In oneembodiment, the user can scale the selected virtual object.Specifically, once the virtual object is selected, the user could use amulti-touch gesture, such as a pinching gesture, to increase or decreasethe size of the virtual object. For example, the size of the selectedvirtual object could decrease when the user brings his two fingersmaking the pinching gesture closer together and could increase when thetwo fingers are moved apart. This scaling could be isotropic (i.e. allthe proportions of the selected virtual object would be maintained) oranisotropic (i.e. the scaling is non-uniform and the selected virtualobject is stretched).

In one embodiment, the selected virtual object can also be moved withinthe 3D virtual environment, as shown in FIG. 12. The movement caninclude translation of the selected virtual object and/or rotation ofthe selected virtual object 1000.

In one embodiment, the selected virtual object 1000 can also beduplicated. Specifically, the system 100 could be provided with acopy-and-paste function to allow one or more copies of the selectedvirtual object to be created in the 3D virtual environment 104.

In one embodiment, an existing virtual object can also be imported intothe 3D virtual environment 104. In this embodiment, the imported virtualobject could be positioned and oriented within the 3D virtualenvironment 104 according to the position and orientation of the planarreference frame 112. To import an existing virtual object, the userwould therefore position and orient the planar reference frame 112according to a desired position and orientation of the imported virtualobject in the 3D virtual environment 104, and then import the virtualobject in the 3D virtual environment. Alternatively, the user may importthe virtual object in the 3D virtual environment 104 first, and thenposition and orient the imported virtual object in the 3D virtualenvironment as desired.

In one embodiment, a pre-existing two-dimensional drawing can also beimported and directly inserted in the 3D virtual environment on theplanar reference 112. The pre-existing two-dimensional drawing could bea vector drawing, a bitmap image, an animation or even a digital videowhich would be could be played in the 3D virtual environment.

In one embodiment, the portable input device 108 further comprises asecondary display for displaying the 3D virtual environment 104. In anembodiment in which the planar input surface comprises a touchscreen,the touchscreen could further define the secondary display, for example.This secondary display could be used to provide additional visualinformation about the 3D virtual environment 104 to the user.

In one embodiment, the secondary display is adapted to display anorthogonal projection of the 3D virtual environment 104 as projected onthe planar reference frame 112, as illustrated in FIG. 13. It will beappreciated that this could provide visual cues to assist the user indrawing on the planar input surface 110.

In an embodiment in which the planar reference frame 112 may be moved inthe hinged movement mode, the orthogonal projection displayed on thesecondary display may be inverted (i.e. flipped by 180 degrees) when theplanar reference frame 112 is rotated about a horizontal axis beyond ahorizontal orientation. This would prevent the secondary display fromdisplaying an upside-down orthogonal projection of the 3D virtualenvironment.

Alternatively, considering that the display unit 106 is adapted todisplay the 3D virtual environment 104 in a perspective view from a mainpoint of view, the secondary display could be adapted to display the 3Dvirtual environment 104 in a perspective view from a secondary point ofview which is different from the main point of view. This configurationcould allow the user to simultaneously observe different elements of the3D virtual environment 104 at once, or to observe the same elements ofthe 3D virtual environment 104, but from different angles.

In one embodiment, the planar reference frame 112 could define across-section plane, such that the secondary display shows across-section of a virtual object when the planar reference frameintersects said virtual object, as best shown in FIG. 11.

Alternatively, the plurality of viewing modes could include any otherviewing mode known to a skilled addressee.

In one embodiment, the processing unit 102 is further adapted forallowing the user to modify at least one of the position and orientationof the main point of view.

For example, the position of the main point of view may be moved bysliding a finger on the planar input surface 110. If the planar inputsurface 110 is horizontal, this will simply move the main point of viewhorizontally within the 3D virtual environment. If the planar inputsurface is oriented at an angle relative to the horizontal when thefinger is sliding on the planar input surface, the main point of viewwill be raised or lowered as it moves laterally. In this case, the rateof elevation will depend on the orientation of the planar input surfacerelative to the horizontal. The main point of view can also be rotatedby using one or more fingers to make a rotation gesture on the planarinput surface 110.

In one embodiment, the movement of the main point of view may be orbitedaround a desired point or a desired vertical axis in the 3D virtualenvironment. Specifically, using one or more fingers to make a rotationgesture on the planar input surface 110 would cause the point of view tomove along an arcuate path having this desired point or vertical axis asits center, while remaining oriented towards (i.e. pointed at) thedesired point or vertical axis. For example, the desired vertical axiscould be defined by a central vertical axis of the planar referenceframe 112. Alternatively, the central vertical axis could be selected bythe user at any desired location within the 3D virtual environment.

In one embodiment, the main point of view may also be selectively tiltedupwardly and downwardly. Specifically, the main point of view may begenerally vertical (i.e. pointed at the horizon) during standardoperation of the system 100. The main point of view could then be tiltedselectively tilted upwardly and downwardly by activating a tilt mode, inwhich rotating the planar input surface 110 about a horizontal axis(i.e. tilting up the planar input surface 110) would rotate the mainpoint of view upwardly or downwardly such that the main point of view ispointed at a location located above or below the horizon, respectively.In one embodiment, the main point of view could then be automaticallyreturned to a generally vertical point of view when the tilt mode isdeactivated.

In one embodiment, the main point of view could be moved and/or rotatedin response to an input from the user and the movement and rotation ofthe main point of view could further continue in a decelerating fashionwhen the user ceases the input. For example, the input could compriseone of the touch gestures described above (sliding gesture and rotationgesture), and the movement of the main point of view could continuewhile the speed of the movement decreases at a certain deceleration rateafter the touch gesture is release, until the movement ends. Thedeceleration rate could be determined, or could be computed based on thespeed at which the main point of view was moving or rotating when thetouch gesture was released. This may contribute in reducing the numberof touch gestures required when the user wishes to move the main pointof view along a relatively large distance within the 3D virtualenvironment.

In one embodiment, there is further provided one or more positioningassistance mechanisms adapted to provide visual cues to the user tofacilitate the positioning of the planar reference frame 112 within the3D virtual environment.

Specifically, when the planar reference frame 112 is moved or rotated,the main point of view could be automatically moved along a horizontalarcuate path while remaining oriented so as to point towards a desiredpoint or vertical axis, for example a central vertical axis of theplanar reference frame 112. Movement of the main point of view couldcomprise relatively slow and small oscillations which would provide a“parallax effect” to enhance the 3D effect of the 3D virtual environment104 as displayed on the display unit 106.

In one embodiment, the planar reference frame 112 is transparent whenidle and becomes semi-translucent when moved or rotated. When the planarreference frame 112 in a semi-translucent state is located between themain point of view and a virtual object, the virtual objects will appearin a darker shade than it normally would. This provides an indication tothe user of the location in the 3D virtual environment 104 of the planarreference frame 112 relative to the virtual object.

In one embodiment, the portable input device 108 further comprisesvarious additional functions, such as control of the lighting positionand orientation within the 3D virtual environment 104.

It will be appreciated that the system 100 described hereinabove maygreatly simplify interactions within the 3D virtual environment 104, ascompared to a conventional interaction system. For example, in a singlegesture, the user could move a virtual object along a three-dimensionalpath within the 3D virtual environment 104 and also rotate and/or scalethe object in the 3D virtual environment. A similar operation in aconventional 3D design program using a cursor and a mouse would requiremultiple steps, including switching between a virtual rotation tool anda virtual translation tool, as well as multiple point of view changes.

In one embodiment, the processing unit 102 comprises a main programwhich generates the 3D virtual environment 104 and the portable inputdevice 108 comprises a secondary program which is operatively connectedto the main program and which receives input from the user and transmitthe received inputs to the main program. Alternatively, the processingunit could be provided within the portable input device 108, and theportable input device 108 could comprises a single program which bothgenerates the 3D virtual environment 104 and receives inputs from theuser. It will be understood that this embodiment is possible as long asthe portable input device 108 has sufficient computing power to generatea desired 3D virtual environment.

It will also be appreciated that the system 100 is particularly welladapted to be used for collaborative interactions within the 3D virtualenvironment. In one embodiment, the system 100 comprises one or moreadditional portable input devices operatively connected to theprocessing unit 104, each one being associated with a correspondingadditional planar reference frame provided in the 3D virtual environment104. Alternatively, a plurality of processing units could be connectedover a network such as an IP-based network, each processing unitallowing access to a common 3D virtual environment.

The embodiments of the invention described above are intended to beexemplary only. The scope of the invention is therefore intended to belimited solely by the scope of the appended claims.

We claim:
 1. A 3D virtual environment interaction system comprising: aprocessor that generates a 3D virtual environment, the 3D virtualenvironment comprising a planar reference frame for allowing a user toperform a user interaction with the 3D virtual environment, the userinteraction including selecting a virtual object in the 3D virtualenvironment by positioning the planar reference frame within the 3Dvirtual environment such that the planar reference frame intersects thevirtual object; a display unit operatively connected to the processorfor displaying the generated 3D virtual environment including the planarreference frame within the 3D virtual environment; a single portableinput device that allows the user to control the position andorientation of the planar reference frame within the generated 3Dvirtual environment, the portable input device comprising a planar inputsurface; a position and orientation sensor for monitoring the positionand orientation of the planar input surface in a real-world space, theposition and orientation sensor being operatively connected to theprocessor to allow the processor to modify at least one of the positionand the orientation of the planar reference frame in response to achange in a corresponding one of the position and orientation of theplanar input surface.
 2. The system as claimed in claim 1, wherein theuser interaction comprises inserting a two-dimensional drawing on theplanar reference frame.
 3. The system as claimed in claim 2, wherein theplanar input surface is adapted to allow the user to draw thetwo-dimensional drawing thereon.
 4. The system as claimed in claim 3,wherein the portable input device comprises at least one virtual drawingtool selected from a group consisting of: a line drawing tool and atwo-dimensional shape drawing tool.
 5. The system as claimed in claim 3,wherein the processor is adapted to produce a three-dimensional drawingwhen the portable device is moved in the real-world space in a directionnormal to the planar input surface, thereby moving the planar referenceframe with the inserted two-dimensional drawing.
 6. The system asclaimed in claim 1, wherein the user interaction further comprisesscaling the selected virtual object.
 7. The system as claimed in claim6, wherein the scaling comprises isotropic scaling.
 8. The system asclaimed in claim 6, wherein the scaling comprises anisotropic scaling.9. The system as claimed in claim 1, wherein the user interactionfurther comprises moving the selected object.
 10. The system as claimedin claim 9, wherein moving the selected object comprises at least one ofmoving the selected object in translation and rotating the selectedobject.
 11. The system as claimed in claim 1, wherein the userinteraction further comprises duplicating the selected object.
 12. Thesystem as claimed in claim 1, wherein the user interaction comprisesimporting an existing virtual object into the 3D virtual environment,and positioning and orienting said existing virtual object within the 3Dvirtual environment according to the position and orientation of theplanar reference frame.
 13. The system as claimed in claim 1, whereinthe processor is adapted to move the planar reference frame according toa selected one of at least one frame movement mode in response to amovement of the planar input surface.
 14. The system as claimed in claim13, wherein the at least one frame movement mode comprises a freemovement mode in which the planar reference frame is moved such that theposition and orientation of the planar reference frame within the 3Dvirtual environment corresponds to the position and orientation of theplanar input surface in the real-world space.
 15. The system as claimedin claim 13, wherein the at least one frame movement mode comprises aplanar movement mode in which movement of the planar reference frame isrestricted to translation in a tangential direction relative to theplanar reference frame.
 16. The system as claimed in claim 13, whereinthe at least one frame movement mode comprises a normal movement mode inwhich movement of the planar reference frame is restricted totranslation in a normal direction relative to the planar referenceframe.
 17. The system as claimed in claim 13, wherein the at least oneframe movement mode comprises a hinged movement mode in which movementof the planar reference frame is restricted to rotation about a rotationaxis extending tangentially relative to the planar reference frame. 18.The system as claimed in claim 13, wherein the at least one framemovement mode comprises an angle snapping mode in which the planarreference frame is adapted to move to one of a plurality ofpredetermined orientations when the orientation of the planar referenceframe is within a predetermined angular range from said one of theplurality of predetermined orientations.
 19. The system as claimed inclaim 18, wherein the predetermined angular range comprises a 5-degreerange.
 20. The system as claimed in claim 1, wherein the planar inputsurface is further adapted to receive at least one touch gesture andwherein the processor is adapted to modify at least one of the positionand orientation of the planar reference frame in response to said atleast one touch gesture.
 21. The system as claimed in claim 20, whereinthe at least one touch gesture comprises a multi-touch gesture.
 22. Thesystem as claimed in claim 1, wherein the portable input device furthercomprises a secondary display for displaying the 3D virtual environment.23. The system as claimed in claim 22, wherein the secondary display isadapted to display an orthogonal projection of the 3D virtualenvironment as projected on the planar reference frame.
 24. The systemas claimed in claim 22, wherein the display unit is adapted to displaythe 3D virtual environment in a perspective view from a main point ofview and the secondary display is adapted to display the 3D virtualenvironment in a perspective view from a secondary point of view whichis different from the main point of view.
 25. The system as claimed inclaim 1, wherein the planar input surface comprises a touchscreen, thetouchscreen further defining the secondary display.
 26. The system asclaimed in claim 1, wherein the display unit is adapted to display the3D virtual environment from a main point of view, and further whereinthe processor is adapted for allowing the user to modify at least one ofthe position and orientation of the main point of view.
 27. The systemas claimed in claim 26, wherein the position of the main point of viewis movable by sliding a finger on the planar input surface.
 28. Thesystem as claimed in claim 27, wherein the planar input surface isadapted to receive a rotation gesture, the processor being adapted tochange the orientation of the main point of view according to saidrotation gesture.
 29. The system as claimed in claim 1, wherein theportable input device comprises a tablet computer having a touchscreenwhich defines the planar input surface.
 30. The system as claimed inclaim 1, wherein the sensor comprises a magnetic sensor including areceiver and an emitter, one of the receiver and the emitter beingattached to the portable input device and the other one of the receiverand the emitter being positioned at a static position within thereal-world space.